AltME groups: search

load-csv fails to deal with these 3 simple (and for me, common) cases:

1,"a
b"
2,"a""b"
3,

>> load-csv %test.csv
== [["1" "a"] [{b"}] ["2" "a" "b"] ["3"]]

I've reverted to an in situ brute force approach:

c: make function! [data /local s] [
		all [find data "|" exit]
		s: false
		repeat i length? trim data [
			switch pick data i [
				#"^""	[s: complement s]
				#","	[all [not s poke data i #"|"]]
				#"^/"	[all [s poke data i #" "]]
			]
		]
		remove-each char data [char = #"^""]

  all [#"|" = last data insert tail data #"|"]	; only required if we're 
  going to parse the data
		parse/all data "|^/"
]

which has 4 minor limitations:

1) the data can't contain the delimter you're going to use ("|" in 
my case)

2) it replaces quoted returns with another character (" " in my code)

3) it removes all quote (") characters (to allow SQLite .import and 
parse/all to function correctly)
4) Individual values are not trimmed (e.g.c "a ,b" -> ["a " "b"])


If you can live with these limitations then the big benefit is that 
you can omit the last two lines and have a string that is import 
friendly for SQLite (or SQL Server) ... this is especially important 
when dealing with large (100MB+) CSV files! ;)

Be careful, if you don't quote string values then the character set 
of your values can't include cr, lf or your delimiter. It requires 
so many changes that it would be more efficient to add new formatter 
functions to the associated FUNCT/with object, then duplicate the 
code in TO-CSV that calls the formatter. Like this:

to-csv: funct/with [
	"Convert a block of values to a CSV-formatted line in a string."
	data [block!] "Block of values"

 /with "Specify field delimiter (preferably char, or length of 1)"
	delimiter [char! string! binary!] {Default ","}
	; Empty delimiter, " or CR or LF may lead to corrupt data
	/no-quote "Don't quote values (limits the characters supported)"
] [
	output: make block! 2 * length? data
	delimiter: either with [to-string delimiter] [","]
	either no-quote [
		unless empty? data [append output format-field-nq first+ data]

  foreach x data [append append output delimiter format-field-nq :x]
	] [
		unless empty? data [append output format-field first+ data]
		foreach x data [append append output delimiter format-field :x]
	]
	to-string output
] [
	format-field: func [x [any-type!] /local qr] [

  ; Parse rule to put double-quotes around a string, escaping any inside

  qr: [return [insert {"} any [change {"} {""} | skip] insert {"}]]
		case [
			none? :x [""]
			any-string? :x [parse copy x qr]
			:x = #"^(22)" [{""""}]
			char? :x [ajoin [{"} x {"}]]
			money? :x [find/tail form x "$"]
			scalar? :x [form x]
			date? :x [to-iso-date x]

   any [any-word? :x binary? :x any-path? :x] [parse to-string :x qr]
			'else [cause-error 'script 'expect-set reduce [

    [any-string! any-word! any-path! binary! scalar! date!] type? :x
			]]
		]
	]
	format-field-nq: func [x [any-type!]] [
		case [
			none? :x [""]
			any-string? :x [x]
			money? :x [find/tail form x "$"]
			scalar? :x [form x]
			date? :x [to-iso-date x]
			any [any-word? :x binary? :x any-path? :x] [to-string :x]
			'else [cause-error 'script 'expect-set reduce [

    [any-string! any-word! any-path! binary! scalar! date!] type? :x
			]]
		]
	]
]


If you want to add error checking to make sure the data won't be 
corrupted, you'll have to pass in the delimiter to format-field-nq 
and trigger an error if it, cr or lf are found in the field data.

Some of what is coming in 2.7.8:

- Bug fixes and enhancements to improve Cheyenne, and other apps 
that have to do similar stuff.

- Some native fixes for non-Windows platforms, particularly Linux.

- Environment variable stuff: GET-ENV expansion on Windows, SET-ENV, 
LIST-ENV

- Function fixes: RUN enabled, LIST-REG/values, possibly TO-LOCAL-FILE

- R2/Forward: FUNCT/extern, LAST?, COLLECT-WORDS, EXTRACT fixes, 
ASCII? fixes, LATIN1? fixes, INVALID-UTF?, CD, LS, MORE, PWD, RM

- (Still pending) Natives: ASSERT, APPLY, RESOLVE, FOREACH set-word 
support

Terry, I think INTERSECT is fine the way it is, and it's easy to 
wrap if you want.

    fold: func [
        series [series!]
        fn     [any-function!]
        /local value
    ][
        value: pick series 1
        foreach item next series [value: fn value item]
    ]

    intersect-all: func [
        series [block!] "A block of series values to intersect"
    ][
        fold series :intersect
    ]

BTW what was the difference if you had:
faced: [area-color: red] ?


In the end the 'area-color was local to each style as well in face/facets. 
In Rebol you cannnot have mixed 'shared' and 'local' values of direct 
type in one object together.

From our expereinece with R3GUI styles, properties that user should 
easily access (like colors etc.) are meant to be usually local to 
each face inastance. That's for what FACES are good enough. If style 
creator think he could 'save' some memory by using shared values 
among faces of the style then he(she) can use the INTERN 'shared 
context' as storage.

Rebolek - I hope you know what you are talking about :-) FACED was 
usefull - it was local instance copy of the value, not the shared 
one. FACETS then kept the values shared. RMA changed the design, 
so that FACETS are now instance local, which of course might lead 
to the increased memory consumption (that would have to be proven 
though). RMA introduced INTERN slot for holding instance local values, 
but from what I saw, that is not used that much yet ...

For unused /local values. For unused refinements I think they must 
be #[none] to be able do code like:
f: func[/switch][ if switch [ true ] ]

This one is a poll question everyone should not have trouble to answer. 
In R2, the USE function initialized the local values to #[unset!] 
for better user protection. In R3 (implementation quirk) the local 
USE variables are initialized to #[none!]. Which alternative do you 
prefer?

I think it should be the same, as if you were to use those values 
local to a function. Aren't they set to #[none] there too?

Good argument, Henrik.

Ladislav, you wrote "the USE function initialized the local values 
to #[unset!] for better user protection."

Better protection than what? Or what is the protection? (Or do you 
just mean USE creating locals is the protection?)

Sorry, improving the formulation:


In R2, the USE function initialized the local values to #[unset!] 
for better user protection against uninitialized variables (variables 
not initialized by the user).

Jerry, the /local refinement is just another function option. If 
you provide that option when you call that function, you can provide 
initial values for the "local variables". In the case of sys/load-module, 
the security and control flow of that function depends on the local 
variables being initialized with the none value, so we want to avoid 
the /local option being used when the function is called. When a 
function refinement is not used its value is none, so if you want 
to ensure that it is none and trigger an error if it isn't, ASSERT/type 
[option none!] is the most efficient way to do this in R3.

Geomol, you can use any other refinement, why use /local and allow 
a huge gap in security wrt intent of a local varable.  

I'd actually replace the /local handling of the function dialect 
and allow you to specify default values right in the function block.

f: func [a b /local i: 2 blk: [ ] ] [...]


The idea of /local is that its a protected, internal, set of words... 
most people don't even realize that it is just another normal refinement.

That last trick would be difficult to do safely though, at least 
as FUNCT is used in the mezzanine generation process. Mezzanines 
are generated with FUNCT but saved with MOLD into the form where 
they will be loaded at runtime. This means that FUNCT can't generate 
code that has inline function or datatype values in it, since they 
won't mold properly. Unless you inline the references to ASSERT and 
NONE!, those words couldn't be used as function parameters or local 
variables in the generated functions. Tradeoffs, I guess.

It's really not any more of a problem for /local than it is for any 
other function option or argument, since the real problem is that 
the techniques for code injection have been revealed. Fortunately, 
so have the methods for avoiding or counteracting it: APPLY, type 
checking, get-words, or wrapping expressions in parens or putting 
them at the end of blocks to make sure that they can't get access 
to modifiable values later on in the block.

just remember that in R3, the /local refinement might be given special 
status in a future release. this was mainly to prevent you from supplying 
default values to locals which can be a pretty big security hole 
right now.

Gregg, about your questions.


1) Has this behavior ever been behind a bug in any of your REBOL 
code? If so, what was the context and what was the impact (e.g., 
how did you work around it)?


I guess, you mean series inside functions. I'm not for everything 
should do a copy, as Gabriele imply. The problem with series in functions 
is only a problem, because functions in REBOL isn't functions in 
the traditional understanding. They are semi-closures. If they were 
functions, the local values would just be on the stack. Because the 
locals live on, this was a problem for me years ago. First I solved 
it by putting COPY in series definitions, then I in many cases changed 
to just do a CLEAR, like in:

local-block: clear []


Now with the discovered behaviour regarding objects, I find the binding 
rules so complicated, that I would forget, how it works in two weeks. 
I will remember, that I have to be very careful, when making objects 
inside functions, especially if it's recursive functions. The consequence 
for me is, that I will probably put COPY/DEEP in, when making objects. 
As I won't start new big projects in REBOL, I will probably not do 
this a lot in practice.

and it's more useful than the other way I think. Once I wrote a function 
to test if two object is similar. It looks a bit silly but works 
for me. Can be extended to test values also:

similar?: func [
    {Returns true if both object has same words in same types.}
    o [object!] p [object!] /local test
][

    test: [if not equal? type? get in o word type? get in p word [return 
    false]]
    foreach word sort first o test
    foreach word sort first p test
    true
]

Everything said about the concurrency model so far indicates that 
it will not be the fork/exec model. Instead it will be the "create 
a new user context from scratch and populate its task-local values 
from the shared stuff" model.

you have no recourse

 is a polite way of saying "you are out of luck". At least regular 
 programmers would be out of luck there - I'm sure someone like Ladislav 
 could come up with an arcane workaround, or you could give up on 
 RETURN and EXIT and use another escape function instead, like THROW. 
 But I assume that you know what I meant by "recourse", and want the 
 point explained.


Pardon me, that question needs some background info. The return models 
are being used to deal with a basic problem of functions catching 
RETURN and EXIT when you don't want them to. This is the case with 
many mezzanine control functions which take and execute a block of 
code. We have been putting requests for new mezzanine control functions 
on hold for quite a while because they can't currently be made to 
pass through RETURN and EXIT, but USE and COLLECT got through before 
we started that, and the restriction is lifted now.


Let's use USE to illustrate, ignoring for the moment that USE *can* 
be rewritten so it doesn't use an inner function.

use: func [
    "Defines words local to a block."
    vars [block! word!] "Local word(s) to the block"
    body [block!] "Block to evaluate"
][
    apply make closure! reduce [to block! vars copy/deep body] []
]


USE uses an inner function to create a binding for its words (the 
closure!). For the dynamic return we have now, the inner function 
catches returns from the body, but even if it didn't the USE function 
itself would catch those returns as well.


One proposal to solve this would be to switch to definitional return, 
which means that RETURN and EXIT would be redefined in the code block 
of a function to return directly to that function, not any intermediate 
function in the call chain. This would solve returns being caught 
by the USE function itself, because the body of code that contains 
the 'return or 'exit words is not physically in the code of the USE 
function, it is only referenced by word. However, that block of code 
is used by the inner function as its code block, so the inner function 
would redefine those words and catch the returns.


If your function uses inner functions like USE does, and can't be 
rewritten to not use them, and you are using definitional return 
without the option to turn it off, then the inner function will localize 
RETURN and EXIT every time.


As a caveat, I wrote that phrase before I came up with the workaround 
in the next section of using direct references to the RETURN and 
EXIT function values instead of referring to them by name, which 
avoids the rebinding issues because no words are involved. See the 
code in http://curecode.org/rebol3/ticket.rsp?id=637to see what 
that workaround makes your code look like.

;-----------------
	;-     swap-values()
	;

 ; given two words, it will swap the values these words reference 
 or contain.
	;-----------------
	swap-values: func [
		'a 'b 
		/local c
	][c: get a set a get b set b  c]


>> a: 5
>> b: 1
>> if a > b [swap-values a b]
>> a
== 1
>> b
== 5


I've been using this to make sure inputs are properly ordered or 
ranges normalized when it matters further down in the code.